[WIP][Cling] Refactor CIFactory and add incremental action support#21903
[WIP][Cling] Refactor CIFactory and add incremental action support#21903SahilPatidar wants to merge 4 commits into
Conversation
Test Results 21 files 21 suites 3d 8h 21m 17s ⏱️ Results for commit 2e181c5. ♻️ This comment has been updated with latest results. |
|
@vgvassilev Some of them show errors like: This may be happening because I made rough changes in Previously, I mentioned an issue related to the Clad plugin consumer. There was a problem with If we allow the plugin consumer, then during In the original Cling implementation, as far as I can see, we never call We need to find a way (to pass plugin into |
|
When clad is OFF we should not run these tests. This means that we need to suppress them in cmake. cc: @guitargeek Are the related llvm/clang changes really required? I think we should do extra effort to avoid them.. |
|
Those are in CodeGenAction.cpp. On my local system, they were causing a segmentation fault. I’m not sure why, but after commenting them out, the crash stopped: And another change related to CompilerInstance and SourceManager initialization. As I mentioned, in Cling we use one large virtual source file to keep source locations consistent. For now, I added a temporary helper: |
Sounds suspicious. Maybe we should run valgrind. |
|
We need to address these issues in the current setup:
For So, I tried overriding the MainFileID inside I also tried another approach (just to test how it behaves). I let BeginSourceFile initialize the SourceManager as usual, and then I created a new virtual file. This file uses the end location of the main file when creating its FileID, so it acts like an includer instead of the main file. const char* Filename = "<<< includer >>>";
FileEntryRef FE = FM.getVirtualFileRef(Filename, 1U << 15U, time(0));
SM.setFileIsTransient(FE);
SourceLocation Result = SM.getLocForEndOfFile(SM.getMainFileID());
m_VirtualFileID =
SM.createFileID(FE, Result, SrcMgr::C_User);
auto Buffer = llvm::MemoryBuffer::getMemBufferCopy("/*CLING DEFAULT MEMBUF*/;\n");
SM.overrideFileContents(FE, std::move(Buffer));After this, I did not see any issue while building. But when I ran roottest-root-tree, a few tests started failing. One of them failed with this error: This error happens inside bool DeclExtractor::CheckForClashingNames(
const llvm::SmallVector<NamedDecl*, 4>& Decls,
DeclContext* DC) {
for (size_t i = 0; i < Decls.size(); ++i) {
...
else if (VarDecl* VD = dyn_cast<VarDecl>(ND)) {
LookupResult Previous(*m_Sema, ND->getDeclName(), ND->getLocation(),
Sema::LookupOrdinaryName,
RedeclarationKind::ForVisibleRedeclaration);
m_Sema->LookupQualifiedName(Previous, DC);
m_Sema->CheckVariableDeclaration(VD, Previous);
...I am not sure why this is happening, and I am also not sure any of these way is correct? Can we do something better? |
|
Can you debug side by side with the working version? |
|
I found the issue. The second approach (using a different virtual file instead of the main file) was failing because of a check in It uses In my case, since I used another virtual file instead of the main file, this function returned false. Because of that, the declaration was not added to the namespace, which caused the name collision error. static bool typedInClingPrompt(FullSourceLoc L) {
if (L.isInvalid())
return false;
const SourceManager &SM = L.getManager();
const FileID FID = SM.getFileID(L);
return SM.isFileOverridden(SM.getFileEntryForID(FID)) &&
(SM.getFileID(SM.getIncludeLoc(FID)) == SM.getMainFileID());
} |
Maybe we can add a check to see if the buffer name starts with Something similar to what we have here: root/core/metacling/src/TCling.cxx Line 1186 in 20705fe |
|
I tried it, and now the tests match the working version on my machine. If second approach is correct to do in Cling, then the only remaining issue is the Clad plugin. |
|
For Clad, here is where we steal the consumers and tune them: https://github.com/vgvassilev/clad/blob/db7f52fafcb45bf8cbaa8b0c7256c664c553e940/tools/ClangPlugin.cpp#L154 and here https://github.com/vgvassilev/clad/blob/db7f52fafcb45bf8cbaa8b0c7256c664c553e940/tools/ClangPlugin.h#L173 I suspect some of these assumptions are incorrect after your patch. |
|
From the code, I see that in the original setup we never called In the original version, we just passed the plugin to the DeclCollector consumer here: In the new version, we pass the plugin consumer in CI during That’s why I directly return the consumer from |
Ok, we will need to think how to circumvent the stealing on clad’s side. Can you think of a way? EDIT: Maybe we steal only when the preprocessor has no CLING macro defined? Or even if we don’t have incremental extensions defined. |
|
I need to understand the behavior when we make changes on the Clad side. How can I modify the Clad plugin and recompile it on root side? |
Not easy. One is edit the source in the build folder clad-prefix, then go to the build directory clad-prefix/…/build and do “make install”. Then cd to the top-most root_build and call “make Cling”. That should be all you need. |
|
I made a few changes to check if this works. First, I commented out Second, I changed this and updated PluginASTAction::ActionType getActionType() override {
// return AddAfterMainAction;
return AddBeforeMainAction;
}In the original setup, we add clad;plugin before the codegen consumer, so I made this change to keep that order. Third, I updated // if (ADecl && !m_Consumer->HandleTopLevelDecl(ADecl.get())) {
if (ADecl && !S.getASTConsumer().HandleTopLevelDecl(ADecl.get())) {After these changes, the tests that were failing on Ubuntu build are now passing. I want to understand how to avoid initialization (so it does not take the consumer), and when we should do that. I don’t see any other test failures locally after these changes: |
|
Clad-side changes - SahilPatidar/clad@f6ad58d. |
Does the clad test suite pass? |
|
I didn’t check on the Clad side. I’ll open a draft PR to see whether it passes Clad-CI. |
|
Currently, we are failing with this kind of error: The build works locally, so it looks like some flag is changing the build behavior in CI, and our changes are triggering it. While building locally, I noticed this flag might be causing the different behavior after our changes: |
That's because the CI was compiling incrementally. I've added a flag to clean build. Can you rebase and push again to trigger a rebuild? |
I mean the build succeeds locally with the command above, but I get the same error if I add these flags: Currently, I am trying to build with the following configuration. If this works, then it seems that the CMAKE_CXX_STANDARD=23 flag is causing some issue with the new changes: |
|
I fixed the issue. It now builds successfully locally with the same configuration. Next, I’ll run ctest. After that, I want to know how to test the Clad-related changes on CI. Can I modify the CMake file in the PR to point to my Clad branch? I mean, just changing this in the PR: |
Yes, exactly. |
|
I've updated the CMake and fix the |
|
Can you resolve the conflicts? |
|
This is coming from recent clad CMake changes. I rebased recently, but later reverted that rebase because while rebuilding ROOT (even clean build) I started getting some build failures from CMake config and some library related issues, like: There was no conflict with my changes. So for now I reverted the rebase to continue build and test the CMake update changes. I can rebase again if this blocks CI build. This CMakeLists change will also revert once we test this on CI. |
|
Without rebasing, can we not run the CI build? |
No, we can’t. The build script rebases against master before running the build and tests. |
|
Can you rebase again to make sure we get a green build? |
|
Hi @smuzaffar, can we check this PR against cmssw? |
|
@vgvassilev , I have started cmssw tests via cms-sw#232 |
| // if (DiagOpts.VerifyDiagnostics) | ||
| // Diags->setClient(new clang::VerifyDiagnosticConsumer(*Diags)); |
There was a problem hiding this comment.
Do we need this? I suspect that this code is used for cling standalone to run the tests. However, maybe with the new changes we don't need it?
| if (CI->getCodeGenOpts().TimePasses) | ||
| CI->createFrontendTimer(); | ||
| // if (CI->getCodeGenOpts().TimePasses) | ||
| // CI->createFrontendTimer(); |
| std::unique_ptr<ASTConsumer> Consumer, | ||
| clang::Preprocessor& PP) { | ||
| m_IncrParser = IncrParser; | ||
| // m_IncrParser = IncrParser; |
There was a problem hiding this comment.
Likewise -- why don't we need that anymore?
| @@ -0,0 +1,99 @@ | |||
| //--------------------------------------------------------------------*- C++ -*- | |||
| // CLING - the C++ LLVM-based InterpreterG :) | |||
| // author: Axel Naumann <axel@cern.ch> | |||
There was a problem hiding this comment.
That's the wrong attribution -- please add your name.
| bool Initialize(llvm::SmallVectorImpl<ParseResultTransaction>& result, | ||
| bool isChildInterpreter); | ||
| clang::CompilerInstance* getCI() const { return m_CI.get(); } | ||
| // clang::CompilerInstance* getCI() const { return m_CI.get(); } |
There was a problem hiding this comment.
We should drop that commented code.
| // // ready before we parse code for the first time. Without C++ modules | ||
| // // we can't setup the calls now because the clang PCH currently just | ||
| // // overwrites it in the Initialize method and we have no simple way to | ||
| // // initialize them earlier. We handle the non-modules case below. |
| # list(APPEND _clad_extra_settings GIT_REPOSITORY https://github.com/vgvassilev/clad.git) | ||
| # list(APPEND _clad_extra_settings GIT_TAG v2.2) | ||
| list(APPEND _clad_extra_settings GIT_REPOSITORY https://github.com/SahilPatidar/clad.git) | ||
| list(APPEND _clad_extra_settings GIT_TAG clad-plugin) |
There was a problem hiding this comment.
Now that change is in master -- so we can route this to vgvassilev/clad.git but the master branch.
This Pull request:
This PR refactors CIFactory and adds support for IncrementalAction (from
Clang-Repl). The goal is to reduce the amount of Clang-specific logic handled inside CIFactory and rely more on Clang’s existing infrastructure.Changes or fixes:
Checklist:
This PR fixes #